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ABSTRACT 

Growing interest in computational prediction of ri- 
bonucleic acid (RNA) three-dimensional structure 
has highlighted the need for reliable and meaning- 
ful methods to compare models and experimental 
structures. We present a structure superposition- 
free method to quantify both the local and global 
accuracy of RNA structural models with respect to 
the reference structure. The method, initially devel- 
oped for proteins and here extended to RNA, closely 
reflects physical interactions, has a simple definition, 
a fixed range of values and no arbitrary parameters. 
It is based on the correspondence of respective con- 
tact areas between nucleotides or their components 
(base or backbone). The better is the agreement be- 
tween respective contact areas in a model and the 
reference structure, the more accurate the model is 
considered to be. Since RNA bases account for the 
largest contact areas, we further distinguish stack- 
ing and non-stacking contacts. We have extensively 
tested the contact area-based evaluation method and 
found it effective in both revealing local discrepan- 
cies and ranking models by their overall quality. Com- 
pared to other reference-based RNA model evalu- 
ation methods, the new method shows a stronger 
emphasis on stereochemical quality of models. In 
addition, it takes into account model completeness, 
enabling a meaningful evaluation of full models and 
those missing some residues. 

INTRODUCTION 

In recent years, the repertoire of known biological functions 
that ribonucleic acid (RNA) performs in the cell has greatly 
expanded (1). Many of these different functions are per- 
formed by RNA molecules or their regions adopting com- 
plex three-dimensional (3D) structures. Not surprisingly, 



the interest in RNA 3D structure has also increased consid- 
erably. However, the determination of RNA 3D structure 
using experimental approaches such as X-ray crystallogra- 
phy or nuclear magnetic resonance remains a formidable 
challenge. Therefore, computational RNA structure predic- 
tion methods are rapidly gaining importance (2). A crit- 
ical component in both the development and compari- 
son of such methods is the ability to evaluate computa- 
tional models against the experimentally determined refer- 
ence structure. Only through the effective reference-based 
model evaluation, one can hope to obtain useful compari- 
son of the performance by different methods. Moreover, the 
quantitative data regarding discrepancies between models 
and corresponding reference RNA structures can provide 
much-needed guidance to methods developers. Therefore, 
the progress in RNA 3D structure prediction is tightly cou- 
pled with the availability of both informative and objective 
scores that quantify discrepancies between modeled and ex- 
perimental structures. 

The best-known score for measuring the differences be- 
tween two 3D structures is root-mean-square deviation 
(RMSD), which reports the average distance between corre- 
sponding pairs of atoms after their optimal superposition. 
However, as an average measure, RMSD is overly sensitive 
to large local errors. One or two poorly modeled residues 
may give a misleading impression about the accuracy of 
the entire model. The recognition of RMSD shortcomings 
has recently led to introduction of several alternative scores. 
Global Distance Test (GDT) (3) is one of the scores in 
the protein field adopted for RNA (4-6). GDT Total Score 
(GDT-TS) calculates the fraction of residues that are within 
1,2,4 and 8 A of the correct position in four independent 
superpositions and reports the average (7). In contrast to 
RMSD, GDT-TS focuses on the most accurate parts of the 
model and is not influenced by outliers. However, the rep- 
resentation of a residue by a single atom (Ca for proteins 
and C3' for RNA), while appropriate for proteins, seems to 
be too coarse-grained for RNA. Moreover, GDT-TS dis- 
tance cutoffs selected to be meaningful for protein models 
may not be optimal for RNA. Several new scores, including 
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Interaction Network Fidelity (INF) (8), Deformation In- 
dex (DI) (8), Deformation Profile (DP) (8) and RNAlyzer 
(9), have been developed specifically for RNA 3D structure. 
INF compares how closely base pairing and base stacking 
interactions within the reference RNA 3D structure are re- 
produced in a model (8). DI is RMSD adjusted by INF and 
has been introduced as an attempt to improve RMSD prop- 
erties on RNA models. The other two new scores, DP and 
RNAlyzer, are also based on RMSD. DP highlights dissim- 
ilarities between a model and the reference structure at the 
nucleotide resolution. RNAlyzer works by comparing how 
well corresponding local neighborhoods in the reference 
structure and a model agree with each other (9). These new 
RN A-specific scores significantly expand the list of available 
model evaluation methods. However, it should be noted that 
they all, except INF, are based on RMSD and, therefore, in- 
herit at least some of its drawbacks. 

In addition to various distance -based approaches, con- 
tacts may also be used to compare models with refer- 
ence structures. Contacts reflect physical interactions in 3D 
structure and even the simplest representation of contacts 
(presence/absence) may be quite informative. Checking not 
only the presence of a contact but also its strength, which re- 
flects the relative interaction intensity, should be even more 
informative. One way to account for the strength of a con- 
tact is to consider its area size (10,11). Recently, apply- 
ing this idea we developed Contact Area Difference score 
(CAD-score), a new method for reference-based protein 
model evaluation (11). CAD-score measures the agreement 
of contact areas for corresponding residues in the refer- 
ence structure and a model. Since interatomic contacts re- 
flect physical interactions (e.g. van der Waals (VDW) inter- 
actions, hydrogen bonds) independently of macromolecule 
type, we decided to explore the feasibility of contact area- 
based approach for RNA. In this study, we report a compar- 
ison of contacts in RNA with those in proteins, the adapta- 
tion of CAD-score for RNA and tests showing the perfor- 
mance of CAD-score on RNA. 

MATERIALS AND METHODS 

Definition of CAD-score 

CAD-score for RNA can be defined in the same way as for 
proteins (11). Let (i,j) denote a contact between nucleotides 
i and j and their contact area in the reference (target) 
structure T. Likewise, let M (iJ ) denote contact area between 
corresponding nucleotides in model M. Now, let G denote 
all the nucleotide-nucleotide contacts in the reference struc- 
ture T. CAD-score quantifying the similarity between ref- 
erence structure T and model M can then be expressed as 
follows: 

r .. r , , E(r,,)eG min ( I T dJ) - ¥f,j) I . T (iJ) ) 
CAD-score = 1 — — 

The summation in the numerator is performed over con- 
tact area differences with their upper bound equal to the 
contact area Tqj) in the reference structure. In other words, 
the contact area difference exceeding is equivalent to 
the missed contact (M (iJ ) = 0). If the nucleotides, present 
in the reference structure, are absent in the model, they are 



treated as if all of their contacts were incorrect. CAD-score 
values are always within the [0, 1] range, with CAD-score 
= 1 indicating identical structures. The equation for CAD- 
score defines global similarity between a model and the ref- 
erence structure. By fixing i the same equation can be used 
to compute a local score for an individual nucleotide. 

Definition and computation of contact areas 

Interatomic contact areas in RNA 3D structure are defined 
and computed in the same way as described previously for 
proteins (11). To derive interatomic contacts within the in- 
put structure, we use the Voronoi tessellation of 3D balls, 
where balls correspond to the heavy atoms of VDW radii 
(12). Two atoms are considered to be in contact with each 
other if they are neighbors in the Voronoi tessellation and 
the water molecule cannot fit between them. For each atom, 
we define a sphere of the radius equal to the sum of VDW 
radius of the atom and the standard radius of the wa- 
ter molecule (1.4 A). The entire surface of this sphere (we 
term it 'contact sphere') is then partitioned into either inter- 
atomic contact areas or solvent accessible areas according 
to the Voronoi tessellation. 

Contact areas between nucleotides are defined by simply 
grouping their interatomic contact areas. The interatomic 
contact areas corresponding to covalent bonds between nu- 
cleotides are not considered. Atomic resolution of contact 
areas makes it possible to define contact areas not only for 
entire nucleotides but also for subsets of nucleotide atoms. 
We use two standard subsets: the sugar-phosphate back- 
bone and the base. This parallels the definition for proteins 
where the main chain and the side chain are considered as 
standard subsets of amino acid residue (11). 

Partitioning of base-base contacts into stacking and non- 
stacking contacts 

To partition base-base contacts into stacking and non- 
stacking ones, we introduced the following definition. Let 
us consider base i, which is in contact with base j, i^j. If all 
atoms (represented as spheres of VDW radii) of base j are 
entirely on one side of the plane of base i, the base-base con- 
tact is defined as the stacking contact. Conversely, if one or 
more atoms (or part of their VDW spheres) of base j appear 
on the other side of the plane of base i than the remaining 
atoms of base j, the contact is defined as non-stacking. The 
illustration of this simple definition is provided in Figure 1. 

CAD-score variants 

In addition to all-atom CAD-score, it is possible to compute 
partial CAD-scores. The consideration of two standard 
subsets of nucleotide atoms, the sugar-phosphate back- 
bone (main chain) and the base (side chain), results in nine 
CAD-score variants, six of them non-redundant (Figure 2). 
Further partitioning of base-base contacts into stacking 
and non-stacking ones results in two additional CAD-score 
variants. 

PDB structure set 

For comparative analysis of contacts in RNA and proteins, 
we used experimentally determined 3D structures that were 
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Figure 1. Illustration of the definition of stacking/non-stacking base- 
base contacts. On the left, a nucleotide in the space-filling representation 
is shown in contact with the three neighbors. Contacts are represented as 
faces of the Voronoi cells constrained by the contact spheres. Cyan and 
magenta indicate stacking and non-stacking contacts, respectively. On the 
right, the same contacts are shown in schematic representation. Molecular 
structures in this and other figures were rendered using Pymol . 
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Figure 2. CAD-score variants based on standard subsets of nucleotide 
atoms. 'A' denotes all atoms, 'S', side chain (base), 'M', main chain (back- 
bone). A non-redundant set is indicated in black color. 



selected from Protein Data Bank (PDB) (as of 1 June 2013). 
The selection included only X-ray structures solved at the 
resolution of 3.0 A or better. In addition, 30% sequence 
identity cutoff was applied to the initial selection to make 
the set non-redundant. 

RNA model test set 

As a test set we used RNA models and corresponding exper- 
imental structures available as part of RNA-puzzles (13), a 
collective experiment for blind RNA structure prediction. 
We used the data of all the challenges completed to date, 
namely 1, 2, 3, 4 and 6. Prior to the analysis, residue num- 
bering and chain identities of raw models were set to match 
the naming of corresponding nucleotides in experimental 
structures. No coordinates of any model were modified. 

RESULTS 

Physical basis of interatomic contacts is the same in both 
protein and RNA 3D structures. We considered that there- 
fore the contact area-based model evaluation score as de- 
fined in its general form (see Materials and Methods) should 
be also feasible for RNA. On the other hand, consider- 
ably different roles of main chain (backbone) and side chain 
(base) atoms in defining secondary and tertiary structures 
in proteins and RNA compelled us to perform a more thor- 
ough investigation of corresponding contacts. 
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Figure 3. Contribution of the three components of all atom-all atom con- 
tacts to the total contact areas in 1 3336 protein (A) and 445 RNA (B) struc- 
tures from PDB. 



Base-base contacts dominate RNA 3D structures 

To investigate the contribution of different types of con- 
tacts in proteins and RNA, we performed the following 
analysis. We selected well-resolved non-redundant protein 
and RNA structures from PDB (see Materials and Meth- 
ods for details). For every structure, we computed the to- 
tal area of all contacts as well as fractions of contact area 
contributed by three types of contacts: (i) main chain- 
main chain (backbone-backbone), (ii) side chain-side chain 
(base-base) and (iii) the remaining contacts that consist 
of side chain-main chain (base-backbone) and main chain- 
side chain (backbone-base) contacts. The results (Figure 3) 
show that, except for the smallest structures, the individual 
contributions to the total contact area by the three con- 
tact types are largely independent of the structure size in 
both proteins and RNA. However, these contributions dif- 
fer significantly in proteins and RNA. In the case of pro- 
teins the contributions by the three types of contacts are 
well-balanced. Although the share of the main chain-main 
chain contacts is the largest (36% on average), the fractions 
of both side chain-side chain and the remaining contacts 
are comparable (correspondingly 30% and 34% on average). 
In the case of RNA the picture is dramatically different. 
Base-base contacts strongly dominate, on average making 
up about half (49%) of all contact areas. These results in- 
dicate that base-base interactions in RNA make a signifi- 
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Figure 4. Dependence of the base-base contact frequency (left) and cu- 
mulative contact areas (right) on the contact area size. The data on all 
base-base contacts (A), base stacking (B) and non-stacking (pairing) (C) 
are shown. Gray bars and lines correspond to contacts determined by the 
approach reported here; red bars and lines correspond to definitions by 
MC-annotate. 



cantly larger impact than side chain-side chain interactions 
in proteins and therefore merit a more detailed analysis. 



Contact area is an effective means for describing base-base 
interactions 

To perform a more detailed analysis of RNA base-base 
contacts, we divided them into bins according to the size 
of contact area. The area size corresponds to the physical 
impact of a contact; therefore, we also looked at the cumu- 
lative impact of contacts (frequency multiplied by the area 
size) for each bin. To compare our results with established 
approaches, for the same set of RNA structures we iden- 
tified base-base interactions using MC-Annotate, a widely 
used RNA annotation method (14). MC-Annotate detects 
and annotates base-base interactions using a procedure 
involving both geometric and probabilistic considerations 
(14,15). Figure 4 shows the comparison of base-base con- 
tact data derived using our approach and MC-Annotate. 
Since MC-Annotate does not compute contact areas, its 
contact data was generated by our approach according to 
the MC-Annotate annotations. If the contact frequency is 



considered (Figure 4A, left), the two approaches show a 
reasonably close agreement, except for the contacts charac- 
terized by small area sizes. Apparently, MC-Annotate does 
not annotate bases as interacting if they barely contact each 
other. If the cumulative area size is considered (Figure 4A, 
right), the agreement is significantly better, since contacts 
with the negligible area size, despite their abundance, con- 
tribute almost nothing to the cumulative impact. One of 
the conclusions that can be made from this comparison is 
that the definition of contacts only as binary information 
(present/absent) may be misleading. A more appropriate 
way is to also consider contact strength, expressed here as 
the contact area size. 



Simple contact-based definition provides a useful approxima- 
tion of base stacking and base pairing 

There are two major types of base-base interactions: base 
stacking and base pairing. Therefore, it would be desirable 
to assign at least approximately base-base contacts to one 
of these two interaction types. We devised an extremely sim- 
ple definition to partition base-base contacts into the two 
types (see Materials and Methods; Figure 1) and applied it 
to the base-base contact data (Figure 4A). If we consider 
undivided base-base contacts, there are three peaks com- 
mon to both the frequency plot (Figure 4A, left) and cumu- 
lative area plot (Figure 4A, right). According to our defi- 
nition, the two rightmost peaks correspond to base stack- 
ing (Figure 4B) while the leftmost of the three peaks cor- 
responds to non-stacking contacts (Figure 4C). To see how 
well this partitioning works, we compared it with the classi- 
fication provided by MC-Annotate. Again, the agreement 
with MC-Annotate improves if the total cumulative con- 
tact area instead of the contact frequency is considered. In 
particular, stacking interactions characterized by the largest 
contact areas agree almost ideally. At the same time even 
for relatively large contact areas there is a visible gap be- 
tween cumulative values of base stacking curves (Figure 
4B). According to our visual analysis at least some of these 
cases can be assigned to either adjacent or non-adjacent 
base stacking interactions (examples are provided in Sup- 
plementary Figure SI). Many other differences represent 
inter-strand base-base overlaps. Although these overlaps 
are not identified as base stacking by MC-Annotate, many 
of them feature fairly large contact areas indicating impor- 
tant contribution to the interaction network. Quite unex- 
pectedly, although non-stacking contacts (Figure 4C) do 
not involve special considerations for hydrogen bonding, 
they very closely recapitulate base pairing interactions de- 
fined by MC-Annotate. 

In the case of unambiguously classified contacts (there 
was an agreement between our approach and MC- 
Annotate), we also looked into the nature of stacked bases 
and the number of hydrogen bonds in base pairs (Sup- 
plementary Figure S2). As might be expected, purine- 
purine stacking dominates the largest contact areas, while 
pyrimidine-pyrimidine stacking is at the lower end of stack- 
ing contact area size. Purine-pyrimidine stacking shows bi- 
modal distribution. As for base pairs, most of them have two 
or three hydrogen bonds. Only a small fraction of contacts, 
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both in numbers and in the cumulative area size, correspond 
to other base pairings. 

Since our approach considers all base-base contacts, 
their division into stacking and non-stacking is, of course, 
oversimplification. However, even this extremely simple 
classification is able to provide a useful distinction between 
most stacking and pairing interactions and thus to reveal 
model errors specific to each interaction type. 

CAD-score combines local contact area differences into the 
global score 

By its nature CAD-score is a local score, as it analyzes dis- 
crepancies only within the immediate 3D neighborhood. 
The most inclusive CAD-score variant quantifies discrepan- 
cies that involve entire nucleotides by taking into account all 
atom-all atom contacts. Additionally defined partial CAD- 
scores measure other types of discrepancies by consider- 
ing contacts between various sets of nucleotide atoms (all 
atoms, backbone or base) or even different types of base- 
base contacts (stacking, non-stacking). A simple combi- 
nation of the local discrepancies of each kind produces 
a global score that summarizes the overall accuracy of a 
model with respect to the reference structure. 

Different CAD-score variants allow addressing different 
questions. However, for practical applications the variants 
that consider either all atom-all atom or base-base contacts 
appear to be the most useful. The usefulness of all atom- 
all atom CAD-score is understandable, since contacts be- 
tween all atoms represent the most complete description 
of the structure. On the other hand, as we have shown, 
base-base contacts represent the dominant contact fraction 
in the RNA and are largely responsible for its specific 3D 
shape. Therefore, the base-base CAD-score and its partial 
(stacking and non-stacking) scores can be particularly use- 
ful in figuring out the cause of discrepancies between the 
two structures. 

Figure 5 shows an example of the evaluation of both lo- 
cal and global accuracy of two RNA-puzzles models (Chal- 
lenge 3) using major variants of CAD-score. The two mod- 
els are of different accuracy, appropriately reflected by the 
'summarizing' CAD-score values. Furthermore, both the 
local discrepancies and the global accuracy values reveal 
that one of the major reasons of the second model being 
inferior to the first one is poorly modeled non-stacking 
(base pairing) interactions. Often, base stacking and non- 
stacking CAD-score values alone may reveal the source of 
error and indicate whether the errors are confined to spe- 
cific regions or dispersed throughout the modeled structure 
(Supplementary Figure S3). 

CAD-score is an effective model ranking index 

CAD-score efficiently accounts for all the local discrep- 
ancies between a model and the reference structure. The 
question is whether the global score, expressed as a sim- 
ple combination of local errors, is also effective in rank- 
ing models by their overall accuracy. Model ranking is in- 
herently subjective, because of the multiple features that 
have to be assessed simultaneously. On the other hand, to 
be considered effective, a new evaluation score should at 



least roughly agree with the currently used scores. To ana- 
lyze model ranking by CAD-score, we compared it with the 
scores used in the RNA-puzzles experiment (13), namely, 
INF, DI and RMSD. We took all the models generated as 
part of the RNA-puzzles experiment, scored them against 
corresponding reference structures and analyzed how well 
CAD-score correlates with each of the other three scores. 
It turned out that CAD-score correlates best with INF, less 
well with DI and least with RMSD. This order does not de- 
pend on whether we use Pearson's correlation coefficient, 
which assumes the linear relationship between scores, or 
Spearman's ranking correlation coefficient, which makes no 
such assumption. Figure 6 shows the relationship between 
two representative CAD-score variants (all atom-all atom 
and base-base) and INF, DI and RMSD. The correlation 
between CAD-score and INF reaches as high as 0.95 indi- 
cating a good agreement between the two scores. The agree- 
ment with DI and RMSD is worse, but correlation values 
are still fairly high. Diverse models available as part of the 
RNA-puzzles experiment represent an excellent test set, but 
their number is relatively small (104 models for 5 reference 
structures). To make the test more rigorous, we performed 
the same analysis using over 30 000 models (for 67 refer- 
ence structures) of the randstr decoy set (6). Although cor- 
relation coefficients calculated using the randstr decoy set 
are slightly smaller, we obtained the same correlation trend: 
INF > DI > RMSD (Supplementary Table SI, Supplemen- 
tary Figure S4). Thus, overall results of the correlation anal- 
ysis indicate that CAD-score model ranking properties are 
closest to those of INF, reflecting their common focus on 
the similarity of interactions. 

CAD-score favors physical realism of structural models 

Correlation analysis revealed that CAD-score shows a fairly 
close agreement with other scores, INF in particular. How- 
ever, inevitably there are cases when the scores disagree. For 
example, it may be that according to CAD-score, model A is 
more accurate than model B, but according to another score 
it is the opposite. An important question is which score to 
trust in such cases. One way to address this question is to 
consider physical realism of models, the feature that does 
not depend on how closely a model agrees with the refer- 
ence structure (11). The idea is that if we take two scores, 
the score that shows stronger tendency to select physically 
more realistic models as the more accurate ones is likely to 
be more objective. 

We asked how CAD-score compares to the other three 
scores (INF, DI and RMSD) in the light of physical real- 
ism of models. For the assessment of physical realism, we 
used the 'clash score', 'bad angles' and 'bad bonds' as re- 
ported by Molprobity (16), a well-known structure valida- 
tion software suite. We considered one of the two models to 
be more physically realistic if the model was better accord- 
ing to at least one of the three Molprobity scores and the 
other two scores did not contradict that (for example, the 
'clash score' was lower, while the values of 'bad bonds' and 
'bad angles' were identical). To perform this analysis, we 
used the RNA-puzzles data set. For every reference struc- 
ture, we compiled all the possible model pairs and identified 
those in which the relative ranking of models was in con- 



5412 Nucleic Acids Research, 2014, Vol. 42, No. 9 



RMSD=15.5A A-A (all to all) S-S (side-chain to side-chain) stacking non-stacking 




Figure 5. Example of CAD-score evaluation of two models of different accuracy at both global and local levels. Different panels show the contact areas 
considered by the indicated CAD-score variants. Contacts are represented as faces of the Voronoi cells constrained by the contact spheres. Blue-white-red 
color gradient represents the accuracy of reproduced contacts (blue — accurate, red — inaccurate). 



flict according to CAD-score and either INF, DI or RMSD. 
We then analyzed the same model pairs with Molprobity. It 
turned out that CAD-score agreed with Molprobity more 
often than did any of the other three scores (Table 1). As 
could be expected by the highest correlation values, the 
smallest number of conflicting rankings was between CAD- 
score and INF. Nevertheless, the support of CAD-score by 
Molprobity was stronger than that of INF In particular, the 
CAD ss -score (evaluating base-base contacts) most closely 
corresponding to INF was supported in about two out of 
three cases. These results show that CAD-score favors phys- 
ical realism of models more strongly than either INF, DI or 
RMSD 



CAD-score accounts for model completeness 

Structural models may not necessarily include all the 
residues. Most often, difficult-to-predict structural regions 
are omitted. A reference-based model evaluation score 
should be able to take this into account properly in order 
to make a fair comparison. 

We asked how well CAD-score, INF, DI and RMSD cope 
with structural models that are heterogeneous as to their 
completeness. To this end, we performed the following anal- 
ysis using RNA-puzzles models. We iteratively truncated 
each model by 20% (removing the equal number of residues 
from both 5' and 3' ends) and recalculated the scores at ev- 
ery step. We monitored the number of models for which the 
score has improved after each truncation step. The idea be- 



hind this test was that if the removed fragment had at least 
some correct features, its removal should make the score 
worse. Even if the removed fragment was completely incor- 
rect, the score of the truncated model should be the same at 
best. 

The results of this test are presented in Table 2. CAD- 
score (both all atom-all atom and base-base contacts) did 
not improve even once upon iterative truncation of models. 
INF has improved seven times, DI — 90 times and RMSD 
-315 times. Thus, it may be concluded that CAD-score is 
suitable for evaluation of a mixture of complete/incomplete 
models. INF is not as good as CAD-score, while DI and 
RMSD could be applied only to models consisting of ex- 
actly the same residues. 

RNA CAD-score on the web 

To make the RNA CAD-score easily accessible, we imple- 
mented it as part of the standalone open-source CAD-score 
software and the corresponding web server (http://www. 
ibt.lt/bioinformatics/CAD-score/) developed previously for 
proteins (11). RNA CAD-score is not limited to the analy- 
sis and comparison of RNA 3D structures. It can be used 
for DNA structures as well. The output of the CAD-score 
calculation for nucleic acids provides both the global simi- 
larity score and local errors in a similar manner as for pro- 
teins. One of the important differences is that for nucleic 
acids, base-base contacts are further subdivided into stack- 
ing and non-stacking contacts. 
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Table 1. Molprobity's "judgment' on model pairs with the conflicting assignment of accuracy by CAD-score and either INF, DI or RMSD 
Model pairs with 

conflicting ranking Supported by Molprobity 



First score Model pairs Second score Model pairs 
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Table 2. The effect of model truncation on evaluation scores 
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DISCUSSION 

Our results show that contact area-based approach can be 
highly effective in quantifying discrepancies between mod- 
eled and reference structures not only for proteins but also 
for RNA. The same general definition of CAD-score can be 
applied to the both types of macromolecules despite their 
significant differences. 

A number of features make CAD-score attractive as a 
similarity measure. First of all, since CAD-score is based on 
comparing contact areas, it does not require structure su- 
perposition. Moreover, contact areas not only define phys- 
ical contacts in the structure, but also indicate their rel- 
ative strength. Therefore, CAD-score reflects physical in- 
teractions that are relevant to the formation and stability 
of 3D structure. The global CAD-score is constructed by 
accounting for all the local discrepancies, thereby provid- 
ing a transparent relationship between local errors and the 
overall model accuracy. Unlike some other scores such as 
RMSD or DI, CAD-score has a fixed value range, simpli- 
fying the comparison of different models. One other attrac- 
tive feature of CAD-score is that its definition does not in- 
volve any arbitrary parameters. In fact, the only adjustable 
parameter used in computing CAD-score is VDW radii of 
heavy atoms. 

In addition to CAD-score based on all atoms, a number 
of partial CAD-score variants can be defined based on sub- 
sets of residue atoms. Since RNA and proteins differ con- 
siderably, we explored the relative impact of contacts con- 
tributed by either main chain (sugar-phosphate backbone) 
or side chain (base). Given the importance of base-base in- 
teractions in the formation and maintenance of both the 
secondary and the tertiary RNA structures it came as no 
surprise that base-base contact areas represent by far the 
largest fraction of all contact areas. Typically, base-base 
interactions are classified into only two types: base stack- 
ing and base pairing. Therefore, we reasoned that it would 
be useful for CAD-score also to have the ability to con- 
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Figure 6. Relationship between CAD-score and INF (A), DI (B) and 
RMSD (C). Data is shown for CADAA-score (left) and CADss-score 
(right). For each plot Pearson's correlation coefficients and Spearman's 
ranking correlation coefficients are indicated. 
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sider these two types of interactions individually. We de- 
vised an extremely simple partitioning of all base-base con- 
tacts into two types (stacking and non-stacking). Our inten- 
tion was not to substitute RNA annotation algorithms but 
rather to provide a useful approximation of the two inter- 
action modes. RNA annotation algorithms are selective in 
defining base stacking and base pairing, while our approach 
takes into consideration all physical contacts. Thus, it was 
surprising to see that our approach and the annotation by 
MC-Annotate show fairly close agreement. It should be em- 
phasized, however, that the agreement is good only when 
the cumulative contact area and not the contact count is 
considered. Perhaps most surprising observation was that 
non-stacking contact areas very closely correspond to base 
pairings defined by MC-Annotate. Since our definition of 
stacking/non-stacking contacts does not involve any special 
treatment of hydrogen bonds, such close agreement suggests 
that the absolute majority of significant non-stacking con- 
tacts originate from base pairs. Disagreement between the 
contact area approach and MC-Annotate largely coincides 
with smaller areas of stacking contacts. Many of these cases 
represent either tiny overlaps of base planes or bases con- 
tacting at an angle and therefore do not represent canonical 
base stacking. However, some large base overlaps ignored 
by MC-Annotate appear to represent typical base stacking, 
suggesting that the definition of stacking in current anno- 
tation algorithms could be improved. The contact area ap- 
proach may help to increase the sensitivity of detecting can- 
didate stacking interactions that subsequently could be re- 
fined using additional criteria. Overall, the analysis of base- 
base interactions suggested that our contact-based defini- 
tion is specific enough to enable CAD-score to focus onto 
discrepancies related to base stacking and base pairing sep- 
arately. 

No matter how a score is defined, its usefulness depends 
entirely on the performance. To make a thorough analysis 
of CAD-score performance, we compared it with the three 
other scores, INF, DI and RMSD, used for the model as- 
sessment during the first round of the RNA-puzzles exper- 
iment (13). We made a comparison of scores according to 
their model ranking properties, the preference of physical 
realism and the ability to take into account model complete- 
ness. These tests revealed that, according to the overall be- 
havior, CAD-score is most similar to INF, less so to DI and 
least similar to RMSD. Taking into account that DI was de- 
signed as an attempt to improve RMSD properties (8), the 
trend of CAD-score agreement with other scores is exactly 
what should be expected from an effective score. The similar 
behavior of CAD-score and INF should not be surprising, 
since both are assessing local interactions. However, despite 
the strong correlation between these two scores, CAD-score 
appears to be superior. 

Firstly, CAD-score shows a stronger preference towards 
more physically realistic models than INF. We believe that 
this is an important property since the improvement accord- 
ing some reference -dependent score should not come at the 
expense of stereochemical quality, which is the reference- 
independent property. The stronger emphasis on physical 
reality by CAD-score might be due to the fact that CAD- 
score takes into account all physical contacts, while INF 
uses only selected set of interactions defined by the structure 



annotation. Furthermore, CAD-score takes into account 
contact strength. The penalty for missed contact depends 
on its area size. Missing important contacts (large contact 
area) is penalized strongly, while missing contacts with neg- 
ligible contact area has almost no effect on the score. In con- 
trast, INF considers only the presence or absence of inter- 
actions, without taking into account how important they 
are. 

Secondly, CAD-score is able to properly account for the 
absence of nucleotides or their parts in a model. Although 
INF, unlike DI or RMSD, shows similar trend, it is not en- 
tirely consistent. When all the evaluated models are com- 
plete this feature has no bearing on model comparison. 
However, if models generated by different methods are com- 
pared, some heterogeneity of model completeness might be 
expected. In such cases the ability to account for missing 
regions would be important. 

In summary, we believe that the attractive properties of 
CAD-score relevant to the RNA 3D structure make CAD- 
score an important addition to the reference -based RNA 
structure evaluation methods. Moreover, taking into ac- 
count the applicability of the method to both nucleic acids 
and protein 3D structures, CAD-score offers new capabili- 
ties for the assessment of 3D structural models of protein- 
nucleic acid complexes. 
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